{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 手写数字识别\n",
    "## 授课大纲\n",
    "* 概要\n",
    "\t* 任务简介\n",
    "    * 构建手写数字识别模型的整体逻辑\n",
    "* MNIST 数据集简介\n",
    "* 导入环境\n",
    "* 数据准备\n",
    "\t* 读入数据\n",
    "    * 数据划分\n",
    "    * 数据乱序\n",
    "    * 数据校验\n",
    "    * 异步读取\n",
    "* 基于前馈神经网络实现手写数字识别任务\n",
    "\t* 神经元\n",
    "\t* 激活函数\n",
    "    * 损失函数\n",
    "\t* 模型构建\n",
    "    * 模型训练\n",
    "    * 模型保存\n",
    "    * 模型评估\n",
    "* 模型优化\n",
    "\t* 学习率\n",
    "    * 激活函数\n",
    "    * 多层前馈神经网络\n",
    "    * 正则化\n",
    "* 可视化分析\n",
    "* 资源配置\n",
    "\n",
    "## 概要\n",
    "### 手写数字识别任务\n",
    "数字识别是计算机从纸质文档、照片或其他来源接收、理解并识别可读的数字的能力，目前比较受关注的是手写数字识别。手写数字识别是一个典型的图像分类问题，已经被广泛应用于汇款单号识别、手写邮政编码识别等领域，大大缩短了业务处理时间，提升了工作效率和质量。\n",
    "\n",
    "在处理如 **图1** 所示的手写邮政编码的简单图像分类任务时，可以使用基于MNIST数据集的手写数字识别模型。MNIST 是深度学习领域标准、易用的成熟数据集，包含 50000 条训练样本和 10000 条测试样本。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/04ab1f9d699e40659a4b69ee069a5136a15cb04bb9d848c2be536da68a8abe5e\" width=\"800\"></center>\n",
    "<center><br>图1：手写数字识别任务示意图</br></center>\n",
    "\n",
    "* 任务输入：一系列手写数字图片，其中每张图片都是28x28的像素矩阵。\n",
    "* 任务输出：经过了大小归一化和居中处理，输出对应的0~9的数字标签。\n",
    "\n",
    "### 构建手写数字识别模型的整体逻辑\n",
    "使用飞桨完成手写数字识别模型任务的代码结构如 **图2** 所示，与使用飞桨完成房价预测模型任务的流程一致，下面的章节中我们将详细介绍每个步骤的具体实现方法和优化思路。\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/38b467ff3b6e4705b9b4d34c6b13431073e449640ef847f396923475b11c913b\" width=\"800\" hegiht=\"\" ></center>\n",
    "<center><br>图2：使用飞桨框架构建神经网络过程</br></center>\n",
    "<br></br>\n",
    "\n",
    "## MNIST 数据集简介\n",
    "MNIST 数据集是从 NIST 的 Special Database 3（SD-3）和 Special Database 1（SD-1）构建而来。Yann LeCun 等人从 SD-1 和 SD-3 中各取一半数据作为 MNIST 训练集和测试集，其中训练集来自 250 位不同的标注员，且训练集和测试集的标注员完全不同。\n",
    "\n",
    "MNIST 数据集的发布，吸引了大量科学家训练模型。1998年，LeCun 分别用单层线性分类器、多层感知器（Multilayer Perceptron, MLP）和多层卷积神经网络 LeNet进行实验，使得测试集的误差不断下降（从12%下降到0.7%）。在研究过程中，LeCun 提出了卷积神经网络（Convolutional Neural Network，CNN），大幅度地提高了手写字符的识别能力，也因此成为了深度学习领域的奠基人之一。\n",
    "\n",
    "手写数字识别的模型是深度学习中相对简单的模型，非常适用初学者。正如学习编程时，我们输入的第一个程序是打印“Hello World！”一样。 在飞桨的入门教程中，我们选取了手写数字识别模型作为启蒙教材，以便更好的帮助读者快速掌握飞桨平台的使用。\n",
    "\n",
    "## 导入环境\n",
    "在正式开始实验之前，让我们看看学习本章内容需要使用到的 python 模块。\n",
    "* os 模块：主要用于处理文件和目录，比如：获取当前目录下文件，删除指定文件，改变目录，查看文件大小等；\n",
    "* json 模块：用于解析 json 格式的文件；\n",
    "* gzip 模块：用于解压以 gz 结尾的文件；\n",
    "* numpy 模块：是常用的科学计算库；\n",
    "* random 模块：是 python 自身的随机化模块，这里主要用于对数据集进行乱序操作；\n",
    "* time 模块：主要用于处理时间系列的数据，在该实验主要用于返回当前时间戳，计算脚本每个epoch运行所需要的时间；\n",
    "* paddle 模块：是百度推出的深度学习开源框架；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import gzip\n",
    "import numpy as np\n",
    "import random\n",
    "import time\n",
    "import paddle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 数据准备\n",
    "在工业实践中，我们面临的任务和数据环境千差万别，通常需要自己编写适合当前任务的数据处理程序，一般涉及如下五个环节：\n",
    "* 读入数据\n",
    "* 数据划分\n",
    "* 数据乱序\n",
    "* 数据校验\n",
    "* 异步读取\n",
    "\n",
    "### 读入数据并划分数据集\n",
    "实际应用中，保存到本地的数据存储格式多种多样，如MNIST数据集以json格式存储在本地，其数据存储结构如 **图3** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/7075f5ca75c54e4e8553c10b696913a1a178dad37c5c460a899cd75635cd7961\" width=\"500\" hegiht=\"\" ></center>\n",
    "<center><br>图3：MNIST数据集的存储结构</br></center>\n",
    "<br></br>\n",
    "\n",
    "**data**包含三个元素的列表：train_set、val_set、 test_set，包括50000条训练样本、10000条验证样本、10000条测试样本。每个样本包含手写数字图片和对应的标签。\n",
    "\n",
    "* **train_set（训练集）**：用于确定模型参数。\n",
    "* **val_set（验证集）**：用于调节模型超参数（如多个网络结构、正则化权重的最优选择）。\n",
    "* **test_set（测试集）**：用于估计应用效果（没有在模型中应用过的数据，更贴近模型在真实场景应用的效果）。\n",
    "\n",
    "**train_set**包含两个元素的列表：train_images、train_labels。\n",
    "\n",
    "* **train_images**：[50000, 784]的二维列表，包含50000张图片。每张图片用一个长度为784的向量表示，内容是 $28 \\times 28$ 尺寸的像素灰度值（黑白图片）。\n",
    "* **train_labels**：[50000, ]的列表，表示这些图片对应的分类标签，即0~9之间的一个数字。\n",
    "\n",
    "在本地`./work/datasets`目录下读取文件名称为`mnist.json.gz`的MNIST数据，并拆分成训练集、验证集和测试集，实现方法如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading mnist dataset from ./work/datasets/mnist.json.gz ......\n",
      "mnist dataset load done\n",
      "训练数据集数量:  50000\n",
      "验证数据集数量:  10000\n",
      "测试数据集数量:  10000\n"
     ]
    }
   ],
   "source": [
    "# 声明数据集文件位置\n",
    "datafile = './work/datasets/mnist.json.gz'\n",
    "print('loading mnist dataset from {} ......'.format(datafile))\n",
    "# 加载json数据文件\n",
    "data = json.load(gzip.open(datafile))\n",
    "print('mnist dataset load done')\n",
    "# 读取到的数据区分训练集，验证集，测试集\n",
    "train_set, val_set, eval_set = data\n",
    "\n",
    "# 观察训练集数据\n",
    "imgs, labels = train_set[0], train_set[1]\n",
    "print(\"训练数据集数量: \", len(imgs))\n",
    "\n",
    "# 观察验证集数量\n",
    "imgs, labels = val_set[0], val_set[1]\n",
    "print(\"验证数据集数量: \", len(imgs))\n",
    "\n",
    "# 观察测试集数量\n",
    "imgs, labels = val= eval_set[0], eval_set[1]\n",
    "print(\"测试数据集数量: \", len(imgs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 数据乱序\n",
    "* **训练样本乱序：** 先将样本按顺序进行编号，建立ID集合index_list。然后将index_list乱序，最后按乱序后的顺序读取数据。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "通过大量实验发现，模型对最后出现的数据印象更加深刻。训练数据导入后，越接近模型训练结束，最后几个批次数据对模型参数的影响越大。为了避免模型记忆影响训练效果，需要进行样本乱序操作。\n",
    "\n",
    "------\n",
    "* **生成批次数据：** 先设置合理的 batch size，再将数据转变成符合模型输入要求的 np.array 格式返回。同时，在返回数据时将 Python 生成器设置为``yield``模式，以减少内存占用。\n",
    "\n",
    "在执行如上两个操作之前，需要先将数据处理代码封装成load_data函数，方便后续调用。load_data有三种模式：``train``、``valid``、``eval``，分为对应返回的数据是训练集、验证集、测试集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据集数量:  50000\n"
     ]
    }
   ],
   "source": [
    "imgs, labels = train_set[0], train_set[1]\n",
    "print(\"训练数据集数量: \", len(imgs))\n",
    "# 获得数据集长度\n",
    "imgs_length = len(imgs)\n",
    "# 定义数据集每个数据的序号，根据序号读取数据\n",
    "index_list = list(range(imgs_length))\n",
    "# 读入数据时用到的批次大小\n",
    "BATCHSIZE = 100\n",
    "\n",
    "# 随机打乱训练数据的索引序号\n",
    "random.shuffle(index_list)\n",
    "\n",
    "# 定义数据生成器，返回批次数据\n",
    "def data_generator():\n",
    "    imgs_list = []\n",
    "    labels_list = []\n",
    "    for i in index_list:\n",
    "        # 将数据处理成希望的类型\n",
    "        img = np.array(imgs[i]).astype('float32')\n",
    "        label = np.array(labels[i]).astype('float32')\n",
    "        imgs_list.append(img) \n",
    "        labels_list.append(label)\n",
    "        if len(imgs_list) == BATCHSIZE:\n",
    "            # 获得一个batchsize的数据，并返回\n",
    "            yield np.array(imgs_list), np.array(labels_list)\n",
    "            # 清空数据读取列表\n",
    "            imgs_list = []\n",
    "            labels_list = []\n",
    "\n",
    "    # 如果剩余数据的数目小于BATCHSIZE，\n",
    "    # 则剩余数据一起构成一个大小为len(imgs_list)的mini-batch\n",
    "    if len(imgs_list) > 0:\n",
    "        yield np.array(imgs_list), np.array(labels_list)\n",
    "    return data_generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "打印第一个batch数据的维度:\n",
      "图像维度: (100, 784), 标签维度: (100,)\n"
     ]
    }
   ],
   "source": [
    "# 声明数据读取函数，从训练集中读取数据\n",
    "train_loader = data_generator\n",
    "# 以迭代的形式读取数据\n",
    "for batch_id, data in enumerate(train_loader()):\n",
    "    image_data, label_data = data\n",
    "    if batch_id == 0:\n",
    "        # 打印数据shape和类型\n",
    "        print(\"打印第一个batch数据的维度:\")\n",
    "        print(\"图像维度: {}, 标签维度: {}\".format(image_data.shape, label_data.shape))\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 数据校验\n",
    "在实际应用中，原始数据可能存在标注不准确、数据杂乱或格式不统一等情况。因此在完成数据处理流程后，还需要进行数据校验，一般有两种方式：\n",
    "\n",
    "* 机器校验：加入一些校验和清理数据的操作。\n",
    "* 人工校验：先打印数据输出结果，观察是否是设置的格式。再从训练的结果验证数据处理和读取的有效性。\n",
    "\n",
    "#### 机器校验\n",
    "\n",
    "如下代码所示，如果数据集中的图片数量和标签数量不等，说明数据逻辑存在问题，可使用assert语句校验图像数量和标签数据是否一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "imgs_length = len(imgs)\n",
    "assert len(imgs) == len(labels), \"length of train_imgs({}) should be the same as train_labels({})\".format(len(imgs), len(label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 人工校验\n",
    "\n",
    "人工校验是指打印数据输出结果，观察是否是预期的格式。实现数据处理和加载函数后，我们可以调用它读取一次数据，观察数据的shape和类型是否与函数中设置的一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "打印第一个batch数据的维度，以及数据的类型:\n",
      "图像维度: (100, 784), 标签维度: (100,), 图像数据类型: <class 'numpy.ndarray'>, 标签数据类型: <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "# 声明数据读取函数，从训练集中读取数据\n",
    "train_loader = data_generator\n",
    "# 以迭代的形式读取数据\n",
    "for batch_id, data in enumerate(train_loader()):\n",
    "    image_data, label_data = data\n",
    "    if batch_id == 0:\n",
    "        # 打印数据shape和类型\n",
    "        print(\"打印第一个batch数据的维度，以及数据的类型:\")\n",
    "        print(\"图像维度: {}, 标签维度: {}, 图像数据类型: {}, 标签数据类型: {}\".format(image_data.shape, label_data.shape, type(image_data), type(label_data)))\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 封装数据准备函数\n",
    "上文，我们从读取数据、划分数据集、到打乱训练数据、构建数据读取器以及数据校验，完成了一整套一般性的数据处理流程，下面将这些步骤放在一个函数中实现，方便在神经网络训练时直接调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def load_data(mode='train'):\n",
    "    datafile = './work/mnist.json.gz'\n",
    "    print('loading mnist dataset from {} ......'.format(datafile))\n",
    "    # 加载json数据文件\n",
    "    data = json.load(gzip.open(datafile))\n",
    "    print('mnist dataset load done')\n",
    "   \n",
    "    # 读取到的数据区分训练集，验证集，测试集\n",
    "    train_set, val_set, eval_set = data\n",
    "    if mode=='train':\n",
    "        # 获得训练数据集\n",
    "        imgs, labels = train_set[0], train_set[1]\n",
    "    elif mode=='valid':\n",
    "        # 获得验证数据集\n",
    "        imgs, labels = val_set[0], val_set[1]\n",
    "    elif mode=='eval':\n",
    "        # 获得测试数据集\n",
    "        imgs, labels = eval_set[0], eval_set[1]\n",
    "    else:\n",
    "        raise Exception(\"mode can only be one of ['train', 'valid', 'eval']\")\n",
    "    print(\"训练数据集数量: \", len(imgs))\n",
    "    \n",
    "    # 校验数据\n",
    "    imgs_length = len(imgs)\n",
    "\n",
    "    assert len(imgs) == len(labels), \\\n",
    "          \"length of train_imgs({}) should be the same as train_labels({})\".format(len(imgs), len(labels))\n",
    "    \n",
    "    # 获得数据集长度\n",
    "    imgs_length = len(imgs)\n",
    "\n",
    "    # 定义数据集每个数据的序号，根据序号读取数据\n",
    "    index_list = list(range(imgs_length))\n",
    "    # 读入数据时用到的批次大小\n",
    "    BATCHSIZE = 100\n",
    "    \n",
    "    # 定义数据生成器\n",
    "    def data_generator():\n",
    "        if mode == 'train':\n",
    "            # 训练模式下打乱数据\n",
    "            random.shuffle(index_list)\n",
    "        imgs_list = []\n",
    "        labels_list = []\n",
    "        for i in index_list:\n",
    "            # 将数据处理成希望的类型\n",
    "            img = np.array(imgs[i]).astype('float32')\n",
    "            label = np.array(labels[i]).astype('float32')\n",
    "            imgs_list.append(img) \n",
    "            labels_list.append(label)\n",
    "            if len(imgs_list) == BATCHSIZE:\n",
    "                # 获得一个batchsize的数据，并返回\n",
    "                yield np.array(imgs_list), np.array(labels_list)\n",
    "                # 清空数据读取列表\n",
    "                imgs_list = []\n",
    "                labels_list = []\n",
    "    \n",
    "        # 如果剩余数据的数目小于BATCHSIZE，\n",
    "        # 则剩余数据一起构成一个大小为len(imgs_list)的mini-batch\n",
    "        if len(imgs_list) > 0:\n",
    "            yield np.array(imgs_list), np.array(labels_list)\n",
    "    return data_generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 异步读取\n",
    "上面提到的数据读取采用的是同步数据读取方式。对于样本量较大、数据读取较慢的场景，建议采用异步数据读取方式。异步读取数据时，数据读取和模型训练并行执行，从而加快了数据读取速度，牺牲一小部分内存换取数据读取效率的提升，二者关系如 **图4** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/a5fd990c5355426183a71b95aa28a59f979014f6905144ddb415c5a4fe647441\" width=\"500\" ></center>\n",
    "<center><br>图4：同步数据读取和异步数据读取示意图</br></center>\n",
    "<br></br>\n",
    "\n",
    "* **同步数据读取**：数据读取与模型训练串行。当模型需要数据时，才运行数据读取函数获得当前批次的数据。在读取数据期间，模型一直等待数据读取结束才进行训练，数据读取速度相对较慢。\n",
    "* **异步数据读取**：数据读取和模型训练并行。读取到的数据不断的放入缓存区，无需等待模型训练就可以启动下一轮数据读取。当模型训练完一个批次后，不用等待数据读取过程，直接从缓存区获得下一批次数据进行训练，从而加快了数据读取速度。\n",
    "* **异步队列**：数据读取和模型训练交互的仓库，二者均可以从仓库中读取数据，它的存在使得两者的工作节奏可以解耦。\n",
    "\n",
    "使用飞桨实现异步数据读取只需要两个步骤：\n",
    "1. 构建一个继承paddle.io.Dataset类的数据读取器。\n",
    "2. 通过paddle.io.DataLoader创建异步数据读取的迭代器。\n",
    "\n",
    "首先，创建定义一个继承 paddle.io.Dataset 的类 MnistDataset，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 创建一个类MnistDataset，继承paddle.io.Dataset 这个类\n",
    "# MnistDataset的作用和上面load_data()函数的作用相同，均是构建一个迭代器\n",
    "class MnistDataset(paddle.io.Dataset):\n",
    "    def __init__(self, mode):\n",
    "        datafile = './work/datasets/mnist.json.gz'\n",
    "        data = json.load(gzip.open(datafile))\n",
    "        # 读取到的数据区分训练集，验证集，测试集\n",
    "        train_set, val_set, test_set = data\n",
    "        if mode == 'train':\n",
    "            # 获得训练数据集\n",
    "            imgs, labels = train_set[0], train_set[1]\n",
    "        elif mode == 'val':\n",
    "            # 获得验证数据集\n",
    "            imgs, labels = val_set[0], val_set[1]\n",
    "        elif mode == 'test':\n",
    "            # 获得测试数据集\n",
    "            imgs, labels = test_set[0], test_set[1]\n",
    "        else:\n",
    "            raise Exception(\"mode can only be one of ['train', 'valid', 'eval']\")\n",
    "        # 校验数据\n",
    "        assert len(imgs) == len(labels), \\\n",
    "            \"length of train_imgs({}) should be the same as train_labels({})\".format(len(imgs), len(labels))\n",
    "        self.imgs = imgs\n",
    "        self.labels = labels\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        img = np.array(self.imgs[idx]).astype('float32')\n",
    "        label = np.array(self.labels[idx]).astype('int64')\n",
    "        return img, label\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "在定义完paddle.io.Dataset后，使用[paddle.io.DataLoader](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0-rc1/api/paddle/io/DataLoader_cn.html) API即可实现异步数据读取，数据会由Python线程预先读取，并异步送入一个队列中。\n",
    "\n",
    "> *class* paddle.io.DataLoader(dataset, batch_size=100, shuffle=True, num_workers=2)\n",
    "\n",
    "DataLoader支持单进程和多进程的数据加载方式。当 num_workers=0时，使用单进程方式异步加载数据；当 num_workers=n(n>0)时，主进程将会开启n个子进程异步加载数据。\n",
    "DataLoader返回一个迭代器，迭代的返回dataset中的数据内容；\n",
    "\n",
    "dataset是支持 map-style 的数据集(可通过下标索引样本)， map-style 的数据集请参考 [paddle.io.Dataset](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0-rc1/api/paddle/io/Dataset_cn.html) API。\n",
    "\n",
    "使用paddle.io.DataLoader API以batch的方式进行迭代数据，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [100, 784] [100]\n",
      "1 [100, 784] [100]\n",
      "2 [100, 784] [100]\n"
     ]
    }
   ],
   "source": [
    "# 声明数据加载函数，使用训练模式，MnistDataset构建的迭代器每次迭代只返回batch=1的数据\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "# 使用paddle.io.DataLoader 定义DataLoader对象用于加载Python生成器产生的数据，\n",
    "# DataLoader 返回的是一个批次数据迭代器，并且是异步的；\n",
    "data_loader = paddle.io.DataLoader(train_dataset, batch_size=100, shuffle=True)\n",
    "# 迭代的读取数据并打印数据的形状\n",
    "for i, data in enumerate(data_loader()):\n",
    "    images, labels = data\n",
    "    print(i, images.shape, labels.shape)\n",
    "    if i>=2:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 基于前馈神经网络实现手写数字识别任务\n",
    "\n",
    "前馈神经网络作为最基础的神经网络，一般包含输入层、隐藏层和输出层。每层神经元只和相邻层神经元相连，即每层神经元只接收相邻前序神经层中神经元所传来的信息，只给相邻后续神经层中神经元传递信息。在前馈神经网络中，同一层的神经元之间没有任何连接，后续神经层不向其前序相邻神经层传递任何信息。前馈神经网络是目前应用最为广泛的神经网络之一。\n",
    "\n",
    "在神经网络中包含一下若干概念：\n",
    "* 神经元\n",
    "* 激活函数\n",
    "* 损失函数\n",
    "\n",
    "### 神经元\n",
    "在生物学中，神经元细胞有兴奋和抑制两种状态。大多数神经元细胞在正常情况下处于抑制状态，一旦某个神经元受到刺激并且电位超过一定阈值之后，这个神经元细胞就被激活，处于兴奋状态，并向其他神经元传递信息。基于神经元细胞的结构特性与传递方式，科学家们提出了 MCP 模型。在人工神经网络中，MCP 模型成为人工神经网络中的最基本结构，MCP 模型结构 **图5** 所示。\n",
    "\n",
    "<center><img src=\"https://github.com/lovejing0306/Images/blob/master/Case/mnist/Neurons.jpg?raw=true\" width=\"500\" ></center>\n",
    "<center><br>图5：MCP 模型</br></center>\n",
    "<br></br>\n",
    "\n",
    "给定 n 个二值化（0或1）的输入数据$x_i$（$1\\le i\\le n$）与连接参数$w_i$（$1\\le i\\le n$），MCP 神经元模型对输入数据线性加权求和，然后使用函数 $\\varPhi()$ 将加权累加结果映射为0 或 1，以完成二分类任务：\n",
    "\n",
    "$$\n",
    "y=\\varPhi \\left( \\sum_{i=1}^n{w_ix_i} \\right) \n",
    "$$\n",
    "\n",
    "其中，$w_i$ 为预先设定的连接权重值，用来表示其所对应输入数据对输出结果的影响；$\\varPhi \\left(  \\right)$将输入端数据与连接权重所得线性加权累加结果与预先设定的阈值 $\\theta$ 进行比较，根据比较结果输出 -1 或 1。也就是说，如果线性加权累加结果大于阈值 $\\theta$ ，则神经元细胞处于兴奋状态，向后传递 1 的信息，否则该神经元细胞处于抑制状态而不向后传递信息。\n",
    "\n",
    "### 激活函数\n",
    "将神经网络用于分类任务，需要将输入数据映射到其语义空间，这一过程就是一个复杂的非线性变换。神经网络使用非线性函数作为激活函数，通过对多个非线性函数组合，来实现对输入信息的非线性变换。为了能够使用梯度下降方法来训练神经网络参数，激活函数必须是连续可导的。\n",
    "\n",
    "#### Sigmoid 激活函数\n",
    "sigmoid 函数的数学表达式如下：\n",
    "$$\n",
    " { f }(x)=\\sigma (x)=\\frac { 1 }{ 1+{ e }^{ -x } } \n",
    "$$\n",
    "\n",
    "其导数形式如下：\n",
    "$$\n",
    " { f }^{ ' }(x)=f(x)(1-f(x))\n",
    "$$ \n",
    "\n",
    "函数图形如 **图6**：\n",
    "\n",
    "<center>\n",
    "    <img \n",
    "    src=\"https://github.com/lovejing0306/Images/blob/master/DeepLearning/Skill/ActiveFunction/sigmoid.jpg?raw=true\"\n",
    "    width=512 height=\n",
    "    >\n",
    "    <br>\n",
    "    <div style=\"color:orange; border-bottom: 1px solid #d9d9d9;\n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图6: sigmoid</div>\n",
    "</center>\n",
    "\n",
    "优点\n",
    "* $Sigmoid$ 函数的输出映射在$(0,1)$之间，可视为概率值；\n",
    "* 单调递增，输入范围无限，输出范围有限；\n",
    "* 非线性变化；\n",
    "* 求导容易；\n",
    "\n",
    "缺点\n",
    "* 由于其软饱和性，容易产生梯度消失，导致训练出现问题；\n",
    "* 其输出并不是以 $0$ 为中心的；\n",
    "\n",
    "#### ReLU 激活函数\n",
    "ReLU（rectified linear unit, 修正线性单元）函数是目前使用较为普遍的激活函数，其数学表达式如下：\n",
    "$$\n",
    " f(x)=\\begin{cases} \\begin{matrix} 0 & x<0 \\end{matrix} \\\\ \\begin{matrix} x & x\\ge 0 \\end{matrix} \\end{cases}\n",
    "$$\n",
    "\n",
    "ReLU 函数的导数为：\n",
    "$$\n",
    " { { f }(x) }^{ ' }=\\begin{cases} \\begin{matrix} 0 & x<0 \\end{matrix} \\\\ \\begin{matrix} 1 & x\\ge 0 \\end{matrix} \\end{cases}\n",
    "$$\n",
    "\n",
    "ReLU 的函数图形如 **图7**：\n",
    "\n",
    "<center>\n",
    "    <img \n",
    "    src=\"https://github.com/lovejing0306/Images/blob/master/DeepLearning/Skill/ActiveFunction/relu.jpg?raw=true\"\n",
    "    width=512 height=\n",
    "    >\n",
    "    <br>\n",
    "    <div style=\"color:orange; border-bottom: 1px solid #d9d9d9;\n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图7:ReLU</div>\n",
    "</center>\n",
    "\n",
    "优点\n",
    "* 收敛速度快；\n",
    "* $Sigmoid$ 涉及了很多 $expensive$ 的操作（比如指数），$ReLU$可以更加简单的实现；\n",
    "* 当输入 $x>=0$ 时，$ReLU$ 的导数为常数，这样可有效缓解梯度消失问题；\n",
    "* 当 $x<0$ 时，$ReLU$ 的梯度总是 $0$，提供了神经网络的稀疏表达能力；\n",
    "\n",
    "缺点\n",
    "* $ReLU$ 的输出不是 $zero-centered$；\n",
    "* 神经元坏死现象，某些神经元可能永远不会被激活，导致相应参数永远不会被更新（在负数部分，梯度为$0$）；\n",
    "* 不能避免梯度爆炸问题；\n",
    "\n",
    "####  Softmax 激活函数\n",
    "$softmax$ 函数一般用在多分类问题中，它是对逻辑斯蒂回归的一种推广，也被称为多项式逻辑斯蒂回归模型。$softmax$ 的数学表达式如下：\n",
    "\n",
    "$$softmax(x_i) = \\frac {e^{x_i}}{\\sum_{j=0}^N{e^{x_j}}}, i=0, ..., C-1$$ \n",
    "\n",
    "其中$C$是标签类别个数。\n",
    "\n",
    "<center>\n",
    "    <img \n",
    "    src=\"https://github.com/lovejing0306/Images/blob/master/Case/iris_classification/softmax.png?raw=true\"\n",
    "    width=512 height=\n",
    "    >\n",
    "    <br>\n",
    "    <div style=\"color:orange; border-bottom: 1px solid #d9d9d9;\n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图8:softmax</div>\n",
    "</center>\n",
    "\n",
    "上图是一个三分类模型，从中可见原始输出的三个数字$3$、$1$、$-3$，经过 $softmax$ 层后转变成加和为1的三个概率值 $0.88$、$0.12$、$0$。$softmax$直白来说就是将原来输出是 $3$、$1$、$-3$ 通过 $softmax$ 函数作用，映射成为 $(0,1)$ 的值， 而这些值的累和为 $1$（满足概率的性质），那么我们就可以将它理解成概率， 在最后选取输出结点的时候，就可以选取概率最大（也就是值对应最大的）结点，作为预测结果。\n",
    "\n",
    "### 损失函数\n",
    "由于在本章节中我们的介绍的是手写数字识别任务，该任务是一个分类任务，因此我们这里主要介绍交叉熵损失函数。\n",
    "\n",
    "交叉熵损失函数的设计是基于最大似然思想：最大概率得到观察结果的假设是真的。如何理解呢？举个例子来说，如 **图9** 所示。有两个外形相同的盒子，甲盒中有99个白球，1个蓝球；乙盒中有99个蓝球，1个白球。一次试验取出了一个蓝球，请问这个球应该是从哪个盒子中取出的？\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/13a942e5ec7f4e91badb2f4613c6f71a00e51c8afb6a435e94a0b47cedac9515\" width=\"400\" hegiht=\"\" ></center>\n",
    "<center><br>图9：体会最大似然的思想 </br></center>\n",
    "<br></br>\n",
    "\n",
    "\n",
    "相信大家简单思考后均会得出更可能是从乙盒中取出的，因为从乙盒中取出一个蓝球的概率更高$（P(D|h)）$，所以观察到一个蓝球更可能是从乙盒中取出的$(P(h|D))$。$D$是观测的数据，即蓝球白球；$h$是模型，即甲盒乙盒。这就是贝叶斯公式所表达的思想：\n",
    "\n",
    "$$P(h|D) ∝ P(h) \\cdot P(D|h)$$\n",
    "\n",
    "依据贝叶斯公式，某二分类模型“生成”$n$个训练样本的概率：\n",
    "\n",
    "$$P(x_1)\\cdot S(w^{T}x_1)\\cdot P(x_2)\\cdot(1-S(w^{T}x_2))\\cdot … \\cdot P(x_n)\\cdot S(w^{T}x_n)$$\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "对于二分类问题，模型为$S(w^{T}x_i)$，$S$为Sigmoid函数。当$y_i$=1，概率为$S(w^{T}x_i)$；当$y_i$=0，概率为$1-S(w^{T}x_i)$。\n",
    "\n",
    "------\n",
    "\n",
    "经过公式推导，使得上述概率最大等价于最小化交叉熵，得到交叉熵的损失函数。交叉熵的公式如下：\n",
    "\n",
    "$$ L = -[\\sum_{k=1}^{n} t_k\\log y_k +(1- t_k)\\log(1-y_k)] $$\n",
    "  \n",
    "其中，$\\log$表示以$e$为底数的自然对数。$y_k$代表模型输出，$t_k$代表各个标签。$t_k$中只有正确解的标签为1，其余均为0（one-hot表示）。\n",
    "\n",
    "因此，交叉熵只计算对应着“正确解”标签的输出的自然对数。比如，假设正确标签的索引是“2”，与之对应的神经网络的输出是0.6，则交叉熵误差是$−\\log 0.6 = 0.51$；若“2”对应的输出是0.1，则交叉熵误差为$−\\log 0.1 = 2.30$。由此可见，交叉熵误差的值是由正确标签所对应的输出结果决定的。\n",
    "\n",
    "\n",
    "### 模型构建\n",
    "接下来我们将使用单层的前馈神经网络构建一个手写数字识别模型，并使用该模型的准确率作为 baseline 结果，模型实现代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义mnist数据识别网络结构\n",
    "class MNIST(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(MNIST, self).__init__()\n",
    "        # 定义一层全连接层\n",
    "        self.fc1 = paddle.nn.Linear(in_features=784, out_features=500)\n",
    "        self.act = paddle.nn.Sigmoid()\n",
    "        # self.act = paddle.nn.ReLU()\n",
    "        self.fc2 = paddle.nn.Linear(in_features=500, out_features=10)\n",
    "        self.softmax = paddle.nn.Softmax()\n",
    "\n",
    "    # 定义网络结构的前向计算过程\n",
    "    def forward(self, x):\n",
    "        outputs = self.fc1(x)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc2(outputs)\n",
    "        outputs = self.softmax(outputs)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 模型训练、评估与保存\n",
    "这里我们创建一个 Trainer 类，该类用于执行训练过程，计算模型分类效果以及保存模型。\n",
    "#### 训练\n",
    "train()用于执行整个训练过程；train_epoch()用于执行一个epoch的训练过程；train_step()用于执行一个批次的训练过程；\n",
    "\n",
    "#### 评估\n",
    "准确率是一个直观衡量分类模型效果的指标，为了能够充分的反应不同方法之间的优略，这里我们实现了 val_epoch() 函数来计算模型的分类准确率。\n",
    "\n",
    "#### 保存\n",
    "save()用于保存模型参数，这里调用了paddle的 save() API；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F\n",
    "\n",
    "class Trainer(object):\n",
    "    def __init__(self, model_path, model, optimizer):\n",
    "        self.model_path = model_path   # 模型存放路径 \n",
    "        self.model = model             # 定义的模型\n",
    "        self.optimizer = optimizer     # 优化器\n",
    "\n",
    "    def save(self):\n",
    "        # 保存模型\n",
    "        paddle.save(self.model.state_dict(), self.model_path)\n",
    "\n",
    "    def val_epoch(self, datasets):\n",
    "        self.model.eval()  # 将模型设置为评估状态\n",
    "        acc = list()\n",
    "        for batch_id, data in enumerate(datasets()):\n",
    "            images, labels = data\n",
    "            pred = self.model(images)   # 获取预测值\n",
    "            # 取 pred 中得分最高的索引作为分类结果\n",
    "            pred = paddle.argmax(pred, axis=-1)  \n",
    "            res = paddle.equal(pred, labels)\n",
    "            res = paddle.cast(res, dtype='float32')\n",
    "            acc.extend(res.numpy())  # 追加\n",
    "        acc = np.array(acc).mean()\n",
    "        return acc\n",
    "\n",
    "    def train_step(self, data):\n",
    "        images, labels = data\n",
    "        # 前向计算的过程\n",
    "        predicts = self.model(images)\n",
    "        # 计算损失\n",
    "        loss = F.cross_entropy(predicts, labels)\n",
    "        avg_loss = paddle.mean(loss)\n",
    "        # 后向传播，更新参数的过程\n",
    "        avg_loss.backward()\n",
    "        self.optimizer.step()\n",
    "        self.optimizer.clear_grad()\n",
    "        return avg_loss\n",
    "\n",
    "    def train_epoch(self, datasets, epoch):\n",
    "        self.model.train()\n",
    "        for batch_id, data in enumerate(datasets()):\n",
    "            loss = self.train_step(data)\n",
    "            # 每训练了1000批次的数据，打印下当前Loss的情况\n",
    "            if batch_id % 500 == 0:\n",
    "                print(\"epoch_id: {}, batch_id: {}, loss is: {}\".format(epoch, batch_id, loss.numpy()))\n",
    "\n",
    "    def train(self, train_datasets, val_datasets, epochs):\n",
    "        for i in range(epochs):\n",
    "            self.train_epoch(train_datasets, i)\n",
    "            train_acc = self.val_epoch(train_datasets)\n",
    "            val_acc = self.val_epoch(val_datasets)\n",
    "            print(\"epoch_id: {}, train acc is: {}, val acc is {}\".format(i, train_acc, val_acc))\n",
    "        self.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.312789]\n",
      "epoch_id: 0, batch_id: 500, loss is: [2.1883173]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.9044656]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.8275037]\n",
      "epoch_id: 0, train acc is: 0.6457599997520447, val acc is 0.6516000032424927\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.8717408]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.8278725]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.751034]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.5611895]\n",
      "epoch_id: 1, train acc is: 0.8088399767875671, val acc is 0.8220999836921692\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.7536268]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.6741691]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.6057963]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.6794984]\n",
      "epoch_id: 2, train acc is: 0.8274000287055969, val acc is 0.836899995803833\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.685547]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.6138568]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.6314843]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.6171939]\n",
      "epoch_id: 3, train acc is: 0.833899974822998, val acc is 0.8396000266075134\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.6336762]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.559969]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.5900352]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.6644902]\n",
      "epoch_id: 4, train acc is: 0.8382400274276733, val acc is 0.8432000279426575\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.6162956]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.6673291]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.5819684]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.7981461]\n",
      "epoch_id: 5, train acc is: 0.8405399918556213, val acc is 0.8439000248908997\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.6321249]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.6643293]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.6398401]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.6001998]\n",
      "epoch_id: 6, train acc is: 0.8437600135803223, val acc is 0.8471999764442444\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.704705]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.5937425]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.6334338]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.6861808]\n",
      "epoch_id: 7, train acc is: 0.8440200090408325, val acc is 0.8460000157356262\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.6340742]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.5915345]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.6343706]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.5487639]\n",
      "epoch_id: 8, train acc is: 0.846780002117157, val acc is 0.849399983882904\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.603699]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.5462135]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.5844387]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.7339121]\n",
      "epoch_id: 9, train acc is: 0.8484200239181519, val acc is 0.8503000140190125\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 0.1\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上述模型中的训练配置如下：\n",
    "* epoch:10\n",
    "* lr:0.1\n",
    "* batch_size:32\n",
    "* optimizer:SGD\n",
    "\n",
    "训练 10 轮后模型在验证集的上的准确率为 85.03%。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 模型优化\n",
    "在前面的实验中我们使用单层的前馈神经网络取得了一个 baseline 的结果，模型的分类准确率为 85.03%。接下来我们通过调整学习率、激活函数、设置更多的隐藏层以及加入正则化的方式来给大家演示如何一步步的提升模型的准确率。\n",
    "\n",
    "### 学习率\n",
    "在深度学习神经网络模型中，通常使用标准的随机梯度下降算法更新参数，学习率代表参数更新幅度的大小，即步长。当学习率最优时，模型的有效容量最大，最终能达到的效果最好。学习率和深度学习任务类型有关，合适的学习率往往需要大量的实验和调参经验。探索学习率最优值时需要注意如下两点：\n",
    "\n",
    "- **学习率不是越小越好**。学习率越小，损失函数的变化速度越慢，意味着我们需要花费更长的时间进行收敛，如 **图10** 左图所示。\n",
    "- **学习率不是越大越好**。只根据总样本集中的一个批次计算梯度，抽样误差会导致计算出的梯度不是全局最优的方向，且存在波动。在接近最优解时，过大的学习率会导致参数在最优解附近震荡，损失难以收敛，如 **图2** 右图所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/1e0f066dc9fa4e2bbc942447bdc0578c2ffc6afc15684154ae84bcf31b298d7b\" width=\"500\" hegiht=\"\" ></center>\n",
    "<center><br>图10: 不同学习率（步长过大/过小）的示意图</br></center>\n",
    "<br></br>\n",
    "\n",
    "在训练前，我们往往不清楚一个特定问题设置成怎样的学习率是合理的，因此在训练时可以尝试调小或调大，通过观察 Loss下降的情况或者模型在训练集和验证集上的准确率判断合理的学习率。\n",
    "\n",
    "通过分析单层前馈神经网络+sigmoid的训练过程，发现模型在训练集和验证集上的效果都比较差，初步分析可能是由于训练过程中学习率过小导致的，现在我们通过增大学习率来看看模型的效果是否有所提升。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.2946477]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.7119272]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.6096951]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.6192926]\n",
      "epoch_id: 0, train acc is: 0.8270000219345093, val acc is 0.838699996471405\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.564203]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.5500519]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.6423697]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.6797929]\n",
      "epoch_id: 1, train acc is: 0.8360999822616577, val acc is 0.8446000218391418\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.5447772]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.6620564]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.5597005]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.7301197]\n",
      "epoch_id: 2, train acc is: 0.8452399969100952, val acc is 0.8500000238418579\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.6207557]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.5245409]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.5826328]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.6157444]\n",
      "epoch_id: 3, train acc is: 0.8535400032997131, val acc is 0.8580999970436096\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.6257117]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.6632313]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.598691]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.5238006]\n",
      "epoch_id: 4, train acc is: 0.856440007686615, val acc is 0.8603000044822693\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.5530074]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.5670847]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.489163]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.5263546]\n",
      "epoch_id: 5, train acc is: 0.8618199825286865, val acc is 0.8669000267982483\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.626404]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.5240252]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.5574493]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.5905216]\n",
      "epoch_id: 6, train acc is: 0.8668400049209595, val acc is 0.8697999715805054\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.5664401]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.5913081]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.5238543]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.6018295]\n",
      "epoch_id: 7, train acc is: 0.8707399964332581, val acc is 0.871999979019165\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.5862826]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.6200182]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.5099155]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.5182421]\n",
      "epoch_id: 8, train acc is: 0.953000009059906, val acc is 0.9513999819755554\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.5117184]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.4959494]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.5416476]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.5614176]\n",
      "epoch_id: 9, train acc is: 0.9606000185012817, val acc is 0.9587000012397766\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 1.0\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过将学习率从 0.1 增大至 1.0 之后，发现模型在验证集上的准确率由之前的 85.03% 提升至 95.87%，由此可见我们之前猜测“学习率过小”的想法是正确的。\n",
    "\n",
    "### 激活函数\n",
    "在《基于前馈神经网络实现手写数字识别任务》小节中，我们已经详细分析过了 sigmoid 激活函数和 relu 激活函数，在浅层神经网络中，与 sigmoid 激活函数相比 relu 激活函数最大的优势在于其提供了稀疏表达的能力，这在一定程度上相当于对网络起到了正则化的作用。这里将 relu 替换为 sigmoid 之后的模型结构代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义mnist数据识别网络结构\n",
    "class MNIST(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(MNIST, self).__init__()\n",
    "        # 定义一层全连接层\n",
    "        self.fc1 = paddle.nn.Linear(in_features=784, out_features=500)\n",
    "        self.act = paddle.nn.ReLU()\n",
    "        self.fc2 = paddle.nn.Linear(in_features=500, out_features=10)\n",
    "        self.softmax = paddle.nn.Softmax()\n",
    "\n",
    "    # 定义网络结构的前向计算过程\n",
    "    def forward(self, x):\n",
    "        outputs = self.fc1(x)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc2(outputs)\n",
    "        outputs = self.softmax(outputs)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.3012307]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.7447678]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.6842225]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.5943805]\n",
      "epoch_id: 0, train acc is: 0.8307399749755859, val acc is 0.8314999938011169\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.776561]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.6330607]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.5804596]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.6410818]\n",
      "epoch_id: 1, train acc is: 0.844980001449585, val acc is 0.8445000052452087\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.5953135]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.6094263]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.5781956]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.6458168]\n",
      "epoch_id: 2, train acc is: 0.8498600125312805, val acc is 0.8479999899864197\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.5822909]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.6323045]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.6275837]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.5682716]\n",
      "epoch_id: 3, train acc is: 0.8575999736785889, val acc is 0.8550999760627747\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.632684]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.564323]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.6428123]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.5875932]\n",
      "epoch_id: 4, train acc is: 0.8628000020980835, val acc is 0.8600000143051147\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.593862]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.601528]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.6831603]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.5625947]\n",
      "epoch_id: 5, train acc is: 0.8651599884033203, val acc is 0.8615000247955322\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.62076]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.56798]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.6378865]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.5313413]\n",
      "epoch_id: 6, train acc is: 0.940559983253479, val acc is 0.9430999755859375\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.509038]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.5549381]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.4904059]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.5149864]\n",
      "epoch_id: 7, train acc is: 0.9530400037765503, val acc is 0.9532999992370605\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.5020424]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.4736552]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.5138539]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.4879482]\n",
      "epoch_id: 8, train acc is: 0.9588599801063538, val acc is 0.9589999914169312\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.4755228]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.4947507]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.4691694]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.5278928]\n",
      "epoch_id: 9, train acc is: 0.9625599980354309, val acc is 0.961899995803833\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 0.1\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过将激活函数替换为 relu，模型在验证集上的准确率由之前的 85.03% 提升至 96.18%，由此可见 relu 在一定程度上要优于 sigmoid 激活函数。\n",
    "\n",
    "### 多层前馈神经网络\n",
    "单层前馈神经网络只包括输入层、隐藏层以及输出层，与单层前馈神经网络相比多层前馈神经网络有多个隐藏层，这意味着多层前馈神经网络有更强的表达能力。这里我们以两个隐藏层的前馈神经网络为例，代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义mnist数据识别网络结构，同房价预测网络\n",
    "class MultiMNIST(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(MultiMNIST, self).__init__()\n",
    "        # 定义一层全连接层，输出维度是1\n",
    "        self.fc1 = paddle.nn.Linear(in_features=784, out_features=512)\n",
    "        self.act = paddle.nn.ReLU()\n",
    "        self.fc2 = paddle.nn.Linear(in_features=512, out_features=256)\n",
    "        self.fc3 = paddle.nn.Linear(in_features=256, out_features=10)\n",
    "        self.softmax = paddle.nn.Softmax()\n",
    "\n",
    "    # 定义网络结构的前向计算过程\n",
    "    def forward(self, x):\n",
    "        outputs = self.fc1(x)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc2(outputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc3(outputs)\n",
    "        outputs = self.softmax(outputs)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.296736]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.7456415]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.6584079]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.5470879]\n",
      "epoch_id: 0, train acc is: 0.9096400141716003, val acc is 0.916700005531311\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.618499]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.5540549]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.4781168]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.5316998]\n",
      "epoch_id: 1, train acc is: 0.9342399835586548, val acc is 0.9348999857902527\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.547365]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.4734488]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.521854]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.5706499]\n",
      "epoch_id: 2, train acc is: 0.9439799785614014, val acc is 0.9435999989509583\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.5178621]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.4757226]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.4958832]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.5620227]\n",
      "epoch_id: 3, train acc is: 0.9548599720001221, val acc is 0.9549000263214111\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.4876502]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.4938526]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.4929041]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.4956522]\n",
      "epoch_id: 4, train acc is: 0.9634000062942505, val acc is 0.9602000117301941\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.4641585]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.4758482]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.5360249]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.4736595]\n",
      "epoch_id: 5, train acc is: 0.9706400036811829, val acc is 0.9639000296592712\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.5271909]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.5129491]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.532759]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.5041648]\n",
      "epoch_id: 6, train acc is: 0.9736199975013733, val acc is 0.9664000272750854\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.5278881]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.4617985]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.4630283]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.4920467]\n",
      "epoch_id: 7, train acc is: 0.9758599996566772, val acc is 0.9692999720573425\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.5241182]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.463443]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.503844]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.4616277]\n",
      "epoch_id: 8, train acc is: 0.9761599898338318, val acc is 0.9661999940872192\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.5192909]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.4897316]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.465244]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.4733565]\n",
      "epoch_id: 9, train acc is: 0.9813200235366821, val acc is 0.9710000157356262\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 0.1\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MultiMNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过在单层前馈神经网路中添加一层隐藏层，模型在验证集上的准确率由之前的 96.18% 提升至 97.10%，模型的精度得到了进一步的提升。\n",
    "\n",
    "### 正则化\n",
    "#### 过拟合现象\n",
    "\n",
    "对于样本量有限、但需要使用强大模型的复杂任务，模型很容易出现过拟合的表现，即在训练集上的损失小，在验证集或测试集上的损失较大，如 **图11** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/99b879c21113494a9d7315eeda74bc4c8fea07f984824a03bf8411e946c75f1b\" width=\"400\" hegiht=\"\" ></center>\n",
    "<center><br>图11：过拟合现象，训练误差不断降低，但测试误差先降后增</br></center>\n",
    "<br></br>\n",
    "\n",
    "反之，如果模型在训练集和测试集上均损失较大，则称为欠拟合。过拟合表示模型过于敏感，学习到了训练数据中的一些误差，而这些误差并不是真实的泛化规律（可推广到测试集上的规律）。欠拟合表示模型还不够强大，还没有很好的拟合已知的训练样本，更别提测试样本了。因为欠拟合情况容易观察和解决，只要训练loss不够好，就不断使用更强大的模型即可，因此实际中我们更需要处理好过拟合的问题。\n",
    "\n",
    "#### 导致过拟合原因\n",
    "\n",
    "造成过拟合的原因是模型过于敏感，而训练数据量太少或其中的噪音太多。\n",
    "\n",
    "如**图12** 所示，理想的回归模型是一条坡度较缓的抛物线，欠拟合的模型只拟合出一条直线，显然没有捕捉到真实的规律，但过拟合的模型拟合出存在很多拐点的抛物线，显然是过于敏感，也没有正确表达真实规律。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/53c389bb3c824706bd2fbc05f83ab0c6dd6b5b2fdedb4150a17e16a1b64c243e\" width=\"700\" hegiht=\"\" ></center>\n",
    "<center><br>图12：回归模型的过拟合，理想和欠拟合状态的表现</br></center>\n",
    "<br></br>\n",
    "\n",
    "如**图13** 所示，理想的分类模型是一条半圆形的曲线，欠拟合用直线作为分类边界，显然没有捕捉到真实的边界，但过拟合的模型拟合出很扭曲的分类边界，虽然对所有的训练数据正确分类，但对一些较为个例的样本所做出的妥协，高概率不是真实的规律。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/b5a46f7e0fbe4f8686a71d9a2d330ed09f23bca565a44e0d941148729fd2f7d7\" width=\"700\" hegiht=\"\" ></center>\n",
    "<center><br>图13：分类模型的欠拟合，理想和过拟合状态的表现</br></center>\n",
    "<br></br>\n",
    "\n",
    "#### 过拟合的成因与防控\n",
    "\n",
    "为了更好的理解过拟合的成因，可以参考侦探定位罪犯的案例逻辑，如 **图14** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/34de60a675b64468a2c3fee0844a168d53e891eaacf643fd8c1c9ba8e3812bcc\" width=\"700\" hegiht=\"\" ></center>\n",
    "<center><br>图14：侦探定位罪犯与模型假设示意</br></center>\n",
    "<br></br>\n",
    "\n",
    "**对于这个案例，假设侦探也会犯错，通过分析发现可能的原因：**\n",
    "\n",
    "1. 情况1：罪犯证据存在错误，依据错误的证据寻找罪犯肯定是缘木求鱼。\n",
    "\n",
    "2. 情况2：搜索范围太大的同时证据太少，导致符合条件的候选（嫌疑人）太多，无法准确定位罪犯。\n",
    "\n",
    "**那么侦探解决这个问题的方法有两种：或者缩小搜索范围（比如假设该案件只能是熟人作案），或者寻找更多的证据。**\n",
    "\n",
    "**归结到深度学习中，假设模型也会犯错，通过分析发现可能的原因：**\n",
    "\n",
    "1. 情况1：训练数据存在噪音，导致模型学到了噪音，而不是真实规律。\n",
    "\n",
    "2. 情况2：使用强大模型（表示空间大）的同时训练数据太少，导致在训练数据上表现良好的候选假设太多，锁定了一个“虚假正确”的假设。\n",
    "\n",
    "**对于情况1，我们使用数据清洗和修正来解决。 对于情况2，我们或者限制模型表示能力，或者收集更多的训练数据。**\n",
    "\n",
    "而清洗训练数据中的错误，或收集更多的训练数据往往是一句“正确的废话”，在任何时候我们都想获得更多更高质量的数据。在实际项目中，更快、更低成本可控制过拟合的方法，只有限制模型的表示能力。\n",
    "\n",
    "#### 正则化项\n",
    "\n",
    "为了防止模型过拟合，在没有扩充样本量的可能下，只能降低模型的复杂度，可以通过限制参数的数量或可能取值（参数值尽量小）实现。\n",
    "\n",
    "具体来说，在模型的优化目标（损失）中人为加入对参数规模的惩罚项。当参数越多或取值越大时，该惩罚项就越大。通过调整惩罚项的权重系数，可以使模型在“尽量减少训练损失”和“保持模型的泛化能力”之间取得平衡。泛化能力表示模型在没有见过的样本上依然有效。正则化项的存在，增加了模型在训练集上的损失。\n",
    "\n",
    "飞桨支持为所有参数加上统一的正则化项，也支持为特定的参数添加正则化项。前者的实现如下代码所示，仅在优化器中设置``weight_decay``参数即可实现。使用参数``coeff``调节正则化项的权重，权重越大时，对模型复杂度的惩罚越高。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.3049517]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.6930485]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.5100982]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.5393964]\n",
      "epoch_id: 0, train acc is: 0.843779981136322, val acc is 0.8483999967575073\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.5944157]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.5461731]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.5863192]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.5110033]\n",
      "epoch_id: 1, train acc is: 0.9290199875831604, val acc is 0.9304999709129333\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.5418378]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.5766435]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.5307435]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.5195917]\n",
      "epoch_id: 2, train acc is: 0.9450399875640869, val acc is 0.9463000297546387\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.4880548]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.5822808]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.515212]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.5522676]\n",
      "epoch_id: 3, train acc is: 0.9553800225257874, val acc is 0.9549000263214111\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.5358081]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.5602996]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.5090479]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.52142]\n",
      "epoch_id: 4, train acc is: 0.9623000025749207, val acc is 0.9610999822616577\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.4967786]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.4890162]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.5190563]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.5194018]\n",
      "epoch_id: 5, train acc is: 0.9658799767494202, val acc is 0.9631999731063843\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.5254526]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.5028758]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.4663583]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.4892447]\n",
      "epoch_id: 6, train acc is: 0.9708399772644043, val acc is 0.9678999781608582\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.4812948]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.494941]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.4981585]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.4940218]\n",
      "epoch_id: 7, train acc is: 0.973800003528595, val acc is 0.9710999727249146\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.4747086]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.4980541]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.4958091]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.4882953]\n",
      "epoch_id: 8, train acc is: 0.9736199975013733, val acc is 0.96670001745224\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.4782419]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.4873502]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.4678252]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.5435168]\n",
      "epoch_id: 9, train acc is: 0.9779199957847595, val acc is 0.9718999862670898\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 0.1\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MultiMNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr,\n",
    "                           weight_decay=paddle.regularizer.L2Decay(coeff=5e-4),\n",
    "                           parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过在多层前馈神经网络的训练过程中添加正则化项，模型在验证集上的准确率由之前的 97.10% 提升至 97.18%，模型的精度得到了略微提升。同时模型在训练集和验证集上准确率的差距得到了进一步的缩小，从而证明正则化的有效性。\n",
    "\n",
    "## 可视化分析\n",
    "训练模型时，经常需要观察模型的评价指标，分析模型的优化过程，以确保训练是有效的。飞桨提供了专业的作图工具 VisualDL，VisualDL是飞桨可视化分析工具，以丰富的图表呈现训练参数变化趋势、模型结构、数据样本、高维数据分布等。帮助用户清晰直观地理解深度学习模型训练过程及模型结构，进而实现高效的模型调优，具体代码实现如下。\n",
    "\n",
    "* 步骤1：引入VisualDL库，定义作图数据存储位置（供第3步使用），本案例的路径是“log”。\n",
    "```\n",
    "from visualdl import LogWriter\n",
    "\n",
    "def get_summary_writer(self):\n",
    "\tif self.summary_dir is None:\n",
    "\t\treturn None\n",
    "\telse:\n",
    "\t\treturn LogWriter(self.summary_dir)\n",
    "        \n",
    "def update_summary(self, **kwargs): \n",
    "\tif self.summary_writer is None:\n",
    "\t\tpass\n",
    "\telse:\n",
    "\t\tfor name in kwargs:\n",
    "\t\t\tself.summary_writer.add_scalar(tag=name, step=self.global_step, value=kwargs[name])\n",
    "```\n",
    "* 步骤2：在训练过程中插入作图语句。当每个batch训练完成后，将当前损失作为一个新增的数据点存储到第一步设置的文件中。使用变量 global_step 记录下已经训练的批次数，作为作图的X轴坐标。\n",
    "\n",
    "```python\n",
    "def train_epoch(self, datasets, epoch):\n",
    "\tself.model.train()\n",
    "\tfor batch_id, data in enumerate(datasets()):\n",
    "\t\tloss = self.train_step(data)\n",
    "\t\tself.update_summary(train_loss=loss.numpy())\n",
    "\t\tself.global_step += 1\n",
    "```\n",
    "\n",
    "完整代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 安装VisualDL\n",
    "!pip install --upgrade --pre visualdl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from visualdl import LogWriter\n",
    "import paddle.nn.functional as F\n",
    "\n",
    "\n",
    "class Trainer(object):\n",
    "    def __init__(self, model_path, model, optimizer, summary_dir=None):\n",
    "        self.model_path = model_path\n",
    "        self.summary_dir = summary_dir   # 作图文件存储目录\n",
    "        self.model = model\n",
    "        self.optimizer = optimizer\n",
    "        self.summary_writer = self.get_summary_writer()  # 获取作图对象\n",
    "        self.global_step = 0\n",
    "\n",
    "    def get_summary_writer(self):\n",
    "        \"\"\"实例化作图对象\"\"\"\n",
    "        if self.summary_dir is None:\n",
    "            return None\n",
    "        else:\n",
    "            return LogWriter(self.summary_dir)\n",
    "\n",
    "    def update_summary(self, **kwargs):  \n",
    "        \"\"\"向作图对象中插入语句\"\"\"\n",
    "        if self.summary_writer is None:\n",
    "            pass\n",
    "        else:\n",
    "            for name in kwargs:\n",
    "                self.summary_writer.add_scalar(tag=name, step=self.global_step, value=kwargs[name])\n",
    "\n",
    "    def save(self):\n",
    "        paddle.save(self.model.state_dict(), self.model_path)\n",
    "\n",
    "    def val_epoch(self, datasets):\n",
    "        self.model.eval()\n",
    "        acc = list()\n",
    "        for batch_id, data in enumerate(datasets()):\n",
    "            images, labels = data\n",
    "            pred = self.model(images)\n",
    "            pred = paddle.argmax(pred, axis=-1)  # 取 pred 中得分最高的索引作为分类结果\n",
    "            res = paddle.equal(pred, labels)\n",
    "            res = paddle.cast(res, dtype='float32')\n",
    "            acc.extend(res.numpy())  # 追加\n",
    "        acc = np.array(acc).mean()\n",
    "        return acc\n",
    "\n",
    "    def train_step(self, data):\n",
    "        images, labels = data\n",
    "\n",
    "        # 前向计算的过程\n",
    "        predicts = self.model(images)\n",
    "\n",
    "        # 计算损失\n",
    "        loss = F.cross_entropy(predicts, labels)\n",
    "        avg_loss = paddle.mean(loss)\n",
    "\n",
    "        # 后向传播，更新参数的过程\n",
    "        avg_loss.backward()\n",
    "        self.optimizer.step()\n",
    "        self.optimizer.clear_grad()\n",
    "        return avg_loss\n",
    "\n",
    "    def train_epoch(self, datasets, epoch):\n",
    "        self.model.train()\n",
    "        for batch_id, data in enumerate(datasets()):\n",
    "            loss = self.train_step(data)\n",
    "            # 记录训练损失\n",
    "            self.update_summary(train_loss=loss.numpy())\n",
    "            self.global_step += 1\n",
    "\n",
    "            # 每训练了1000批次的数据，打印下当前Loss的情况\n",
    "            if batch_id % 500 == 0:\n",
    "                print(\"epoch_id: {}, batch_id: {}, loss is: {}\".format(epoch, batch_id, loss.numpy()))\n",
    "\n",
    "    def train(self, train_datasets, val_datasets, epochs):\n",
    "        for i in range(epochs):\n",
    "            self.train_epoch(train_datasets, i)\n",
    "            train_acc = self.val_epoch(train_datasets)\n",
    "            val_acc = self.val_epoch(val_datasets)\n",
    "            self.update_summary(train_acc=train_acc, val_acc=val_acc)\n",
    "            print(\"epoch_id: {}, train acc is: {}, val acc is {}\".format(i, train_acc, val_acc))\n",
    "        self.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.3144398]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.6106814]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.6772048]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.5089396]\n",
      "epoch_id: 0, train acc is: 0.8843200206756592, val acc is 0.8916000127792358\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.6163015]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.5770197]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.6005037]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.4678018]\n",
      "epoch_id: 1, train acc is: 0.9097999930381775, val acc is 0.9133999943733215\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.478539]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.5579653]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.5488136]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.464844]\n",
      "epoch_id: 2, train acc is: 0.9307199716567993, val acc is 0.9320999979972839\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.4766693]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.5465415]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.5130544]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.5368116]\n",
      "epoch_id: 3, train acc is: 0.9395599961280823, val acc is 0.9413999915122986\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.6749122]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.5553331]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.5097637]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.5396996]\n",
      "epoch_id: 4, train acc is: 0.947160005569458, val acc is 0.9492999911308289\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.4831853]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.5456942]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.5167698]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.504236]\n",
      "epoch_id: 5, train acc is: 0.9502400159835815, val acc is 0.949400007724762\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.4614844]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.4634542]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.5861702]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.4884583]\n",
      "epoch_id: 6, train acc is: 0.9536799788475037, val acc is 0.9520000219345093\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.4617974]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.4648235]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.5836354]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.5157912]\n",
      "epoch_id: 7, train acc is: 0.9582800269126892, val acc is 0.9567999839782715\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.5309503]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.5045311]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.4930751]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.4735945]\n",
      "epoch_id: 8, train acc is: 0.9615600109100342, val acc is 0.9588000178337097\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.5004003]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.53256]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.5303413]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.5316935]\n",
      "epoch_id: 9, train acc is: 0.9661200046539307, val acc is 0.9617000222206116\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 1.0\n",
    "model_path = './mnist.pdparams'\n",
    "summary_dir = './work/summary'\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt,\n",
    "    summary_dir=summary_dir\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "* 步骤3：命令行启动VisualDL。\n",
    "\n",
    "使用“visualdl --logdir [数据文件所在文件夹路径] 的命令启动VisualDL。在VisualDL启动后，命令行会打印出可用浏览器查阅图形结果的网址。\n",
    "``` \n",
    "$ visualdl --logdir ./log --port 8080\n",
    "```\n",
    "\n",
    "* 步骤4：打开浏览器，查看作图结果，如下图所示。\n",
    "\n",
    "查阅的网址在第三步的启动命令后会打印出来（如http://127.0.0.1:8080/），将该网址输入浏览器地址栏刷新页面的效果如下图所示。除了右侧对数据点的作图外，左侧还有一个控制板，可以调整诸多作图的细节。\n",
    "\n",
    "\n",
    "<center><img src=\"https://github.com/lovejing0306/Images/blob/master/Case/mnist/VisualDL.jpg?raw=true\" width=\"800\" hegiht=\"\" ></center>\n",
    "<center><br>VisualDL作图示例</br></center>\n",
    "<br></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 资源配置\n",
    "使用飞桨构建的模型可以指定运行的资源，通过 paddle.set_device API，设置在 GPU 上训练还是 CPU 上训练。\n",
    "\n",
    "> paddle.set_device *(device)*\n",
    "\n",
    "参数 \n",
    "device (str)：此参数确定特定的运行设备，可以是`cpu`、 `gpu:x`或者是`xpu:x`。其中，`x`是GPU或XPU的编号。当device是`cpu`时， 程序在CPU上运行；当device是`gpu:x`时，程序在GPU上运行。\n",
    "\n",
    "修改后的代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch_id: 0, batch_id: 0, loss is: [2.2991498]\n",
      "epoch_id: 0, batch_id: 500, loss is: [1.772263]\n",
      "epoch_id: 0, batch_id: 1000, loss is: [1.6144236]\n",
      "epoch_id: 0, batch_id: 1500, loss is: [1.5906253]\n",
      "epoch_id: 0, train acc is: 0.8229399919509888, val acc is 0.8270999789237976\n",
      "epoch_id: 1, batch_id: 0, loss is: [1.6858568]\n",
      "epoch_id: 1, batch_id: 500, loss is: [1.578338]\n",
      "epoch_id: 1, batch_id: 1000, loss is: [1.5622997]\n",
      "epoch_id: 1, batch_id: 1500, loss is: [1.6056569]\n",
      "epoch_id: 1, train acc is: 0.8350200057029724, val acc is 0.8353999853134155\n",
      "epoch_id: 2, batch_id: 0, loss is: [1.7619798]\n",
      "epoch_id: 2, batch_id: 500, loss is: [1.6406751]\n",
      "epoch_id: 2, batch_id: 1000, loss is: [1.5183274]\n",
      "epoch_id: 2, batch_id: 1500, loss is: [1.5920503]\n",
      "epoch_id: 2, train acc is: 0.8440799713134766, val acc is 0.8424999713897705\n",
      "epoch_id: 3, batch_id: 0, loss is: [1.5738984]\n",
      "epoch_id: 3, batch_id: 500, loss is: [1.6155759]\n",
      "epoch_id: 3, batch_id: 1000, loss is: [1.6095839]\n",
      "epoch_id: 3, batch_id: 1500, loss is: [1.5966538]\n",
      "epoch_id: 3, train acc is: 0.8487200140953064, val acc is 0.8472999930381775\n",
      "epoch_id: 4, batch_id: 0, loss is: [1.6647048]\n",
      "epoch_id: 4, batch_id: 500, loss is: [1.6401861]\n",
      "epoch_id: 4, batch_id: 1000, loss is: [1.6518046]\n",
      "epoch_id: 4, batch_id: 1500, loss is: [1.6959736]\n",
      "epoch_id: 4, train acc is: 0.8572999835014343, val acc is 0.8546000123023987\n",
      "epoch_id: 5, batch_id: 0, loss is: [1.585624]\n",
      "epoch_id: 5, batch_id: 500, loss is: [1.5788538]\n",
      "epoch_id: 5, batch_id: 1000, loss is: [1.6181412]\n",
      "epoch_id: 5, batch_id: 1500, loss is: [1.5896239]\n",
      "epoch_id: 5, train acc is: 0.861519992351532, val acc is 0.859000027179718\n",
      "epoch_id: 6, batch_id: 0, loss is: [1.6496923]\n",
      "epoch_id: 6, batch_id: 500, loss is: [1.6539164]\n",
      "epoch_id: 6, batch_id: 1000, loss is: [1.5827528]\n",
      "epoch_id: 6, batch_id: 1500, loss is: [1.6257279]\n",
      "epoch_id: 6, train acc is: 0.8660600185394287, val acc is 0.8628000020980835\n",
      "epoch_id: 7, batch_id: 0, loss is: [1.5875708]\n",
      "epoch_id: 7, batch_id: 500, loss is: [1.5556587]\n",
      "epoch_id: 7, batch_id: 1000, loss is: [1.6409627]\n",
      "epoch_id: 7, batch_id: 1500, loss is: [1.5946726]\n",
      "epoch_id: 7, train acc is: 0.8692399859428406, val acc is 0.8659999966621399\n",
      "epoch_id: 8, batch_id: 0, loss is: [1.5394361]\n",
      "epoch_id: 8, batch_id: 500, loss is: [1.6018511]\n",
      "epoch_id: 8, batch_id: 1000, loss is: [1.609303]\n",
      "epoch_id: 8, batch_id: 1500, loss is: [1.5235153]\n",
      "epoch_id: 8, train acc is: 0.8725000023841858, val acc is 0.8693000078201294\n",
      "epoch_id: 9, batch_id: 0, loss is: [1.6429791]\n",
      "epoch_id: 9, batch_id: 500, loss is: [1.5952787]\n",
      "epoch_id: 9, batch_id: 1000, loss is: [1.554583]\n",
      "epoch_id: 9, batch_id: 1500, loss is: [1.6479154]\n",
      "epoch_id: 9, train acc is: 0.8741999864578247, val acc is 0.8711000084877014\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "lr = 1.0\n",
    "use_gpu = False\n",
    "model_path = './mnist.pdparams'\n",
    "\n",
    "paddle.set_device('gpu:0') if use_gpu else paddle.set_device('cpu')\n",
    "\n",
    "train_dataset = MnistDataset(mode='train')\n",
    "train_loader = paddle.io.DataLoader(train_dataset,\n",
    "                                    batch_size=32,\n",
    "                                    shuffle=True,\n",
    "                                    num_workers=4)\n",
    "\n",
    "val_dataset = MnistDataset(mode='val')\n",
    "val_loader = paddle.io.DataLoader(val_dataset, batch_size=128)\n",
    "\n",
    "model = MNIST()\n",
    "opt = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "trainer = Trainer(\n",
    "    model_path=model_path,\n",
    "    model=model,\n",
    "    optimizer=opt\n",
    ")\n",
    "\n",
    "trainer.train(train_datasets=train_loader, val_datasets=val_loader, epochs=epochs)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
